Chi-square Tests Driven Method for Learning the Structure of Factored MDPs

sdyna is a general framework designed to address large stochastic reinforcement learning (rl) problems. Unlike previous model-based methods in Factored mdps (fmdps), it incrementally learns the structure of a rl problem using supervised learning techniques. spiti is an instantiation of sdyna that uses decision trees as factored representations. First, we show that, in structured rl problems, spiti learns the structure of fmdps using Chi-Square tests and performs better than classical tabular model-based methods. Second, we study the generalization property of spiti using a Chi-Square based measure of the accuracy of the model built by spiti.